In recent years, a pseudo image generating apparatus, that generates by simulation a pseudo image of an image obtained by imaging an object from a virtual viewpoint that is different from a viewpoint from which the object is actually imaged, without actually imaging the object from the virtual viewpoint, has been started to be used for applications for generating stereoscopically viewable stereoscopic images.
In an image processing system of Japanese Patent Application Laid-Open No. 2003-317117, when a corresponding point search is performed among a plurality of two-dimensional images and a pseudo image for three-dimensional display is obtained from distance information obtained based on a result of searching for the corresponding points, an average value of the distance information obtained by success in the search for the corresponding points is obtained as distance information of a region where the distance information cannot be obtained due to a failure of the corresponding point searching so that a pseudo image is restored. Further, the pseudo image is subject to a smoothing process so that a pseudo image whose smoothness is improved is obtained also in the region where the search for the corresponding points fails.